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We investigate the implications for fermion mass models in heterotic orbifolds of 
the modular symmetry mixing twisted states localized at different fixed points. We 
show that, unlike in the case of continuous gauge symmetries, the mass eigenstates 
do not mix under the symmetry; thus coupling constants in the low-energy theory 
are invariant functions of the moduli. 



Introduction 

Twisted sectors of heterotic orbifolds have long been of phenomenologi- 
cal interest, since their massless states are automatically localized at the 
fixed points, resulting in effective couplings exponentially suppressed by 
the distance between different points 1 . In the field theory limit of large 
compactification radii, such nonlocal couplings vanish, but they acquire 
nonzero values through nonperturbative effects on the worldsheet involv- 
ing stretched strings. Attempts have been made to explain the hierarchy 
of fermion masses (which generally requires some Yukawa couplings to be 
orders of magnitude smaller than unity) and the CKM charged current 
mixing of quarks, by using this suppression 2 . 

The usual route to explaining small Yukawa couplings in a natural way is 
a (broken) flavour symmetry, so one might ask whether there is a symmetry 
associated with this suppression of nonlocal interactions. In fact, the target- 
space modular invariance of heterotic strings 3 might play such a role. This 
symmetry can be thought of as the extension of T-duality to a d-dimensional 
torus, where d > 2. In the simplest case d = 2, the path integral for 
strings propagating on a background with modulus T ~ V + iB, where V 
is the torus volume in string units and B is the antisymmetric tensor field 
(Bmn = Bcmn where M, N label the torus coordinates), is equivalent to 
that on a background T', where V = (aT - i(3)/(ijT + 5), a,(3,j,S 6 Z, 
a5 - [3y = 1. This (P)SL(2,Z) symmetry is generated by S: T i-> 1/T, 
T:T i— > T + i. For 6-dimensional orbifolds, the full duality group may be 
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larger but includes three PSL(2,Z) subgroups acting on diagonal moduli 
T 1 , / = 1, . . . , 3. Correlation functions involving twisted states, which are 
related to Yukawa couplings, are then invariant, provided that the states 
in a given sector have the nontrivial transformation 4 

Ui i > U t3 (J 3 , U ] U = 1 (1) 

where U depends on a, (3, 7, and 8 but a not on T. This discrete non- 
abelian symmetry relates states with the same quantum numbers, acting 
as a horizontal symmetry, which may be relevant for the flavour problem. 

Previous investigations of mass textures used models with more than 
three matter generations massless at tree level, and exploited the freedom 
to pick a subset of the fixed point states to correspond to the MSSM matter 
fields, under the assumption that the remaining states became unobserv- 
ably heavy. Physical observables in such a framework are, in general, not 
invariant functions of T since MSSM states transform into heavy states un- 
der some group elements 5 . However, it can be shown that such behaviour 
is inconsistent with modular invariance of the mass term in the Lagrangian, 
since there must be off-diagonal masses in a basis where supposedly "light" 
and "heavy" states are mixed by the symmetry 6 . One can find a basis where 
some elements of SL(2, Z) act diagonally, but other elements inevitably mix 
states within a twisted sector. We find that (except in very special cases) 
the resulting mass eigenstates are SL(2,Z) invariant, T-dcpendent linear 
combinations of the original fixed point states. Thus, even when the modu- 
lar symmetry is spontaneously broken, the couplings of the light states are 
invariant functions of moduli. 

Invariance of mass eigenstates 

Our starting points are the existence of the unitary transformation U and 
the invariance of the part of the d = A effective Lagrangian bilinear in matter 
fields Ci, which will result in mass terms: 

-C m = ^CiCjyiju (T)H k J] + h. c. = CiCjMij (T, H, X) + h. c. (2) 

m 

where i runs over complete twisted sectors' 3 . Here, yijki are T-dependent 
Yukawa couplings, and we include an arbitrary number of Higgses Hk and 



a Up to a overall complex phase within each twisted sector. 

b The d are left-handed Weyl fermions, with My = Mji a complex symmetric matrix: 
this notation allows for both Dirac and Majorana masses. Gauge indices are suppressed. 
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SM singlets X m which may acquire v.e.v.'s in the stabilized vacuum; all 
fields may have nontrivial modular transformation properties. 

To determine whether it is possible to separate the states into two sets, 
light (observable) and heavy, which intermix under SL(2, Z)x, we must find 
the modular transformation of the mass eigenstates, given the matrices U 
acting in the fixed point basis. When general off-diagonal mass matrices 
are considered, such transformations quickly become unwieldy: so instead 
of the fixed point basis, we start with a basis where the modular trans- 
formation that we wish to consider acts as a diagonal matrix of phases: 
r 6 SL(2,Z): C\ i — ► e'^Cj. This can always be done, because unitary ma- 
trices are normal, and does not introduce additional T-dependence. Then, 
since £ m necessarily transforms into itself, we have in this basis 

Mij h- e-^+^M^, V jp i ► e^V jp (3) 

where V is T-, Hk-, and X m -dependent unitary matrix satisfying V T MV = 
d\a.g(mi(T,H,X)) c . Then the mass eigenstates ^ p = V^-Cj are invariant 
when the C'i, T, Hk and X m are all transformed under T. If we rewrite the 
theory in the 'J basis, all coupling constants must be invariant functions of 
T, Hk, and X m , following from the invariance of the 4d effective Lagrangian. 

There might be exceptions to this result if the off-diagonal "light-heavy" 
mass-terms vanish exactly, or when the modular transformation acts on 
the d as a permutation. However, the first possibility could only occur 
at isolated points in moduli space d , and the second is not realised in any 
known orbifold. 

The case of antigenerations which acquire masses by pairing up with 
matter generations at energies somewhat below the string scale can be 
treated by the above method, but one can also think of it in two stages at 
widely-separated mass scales. First, consider a N g + M-by-M-dimensional 
mass matrix, where M is the number of paired (anti) generations, with M 
large mass eigenvalues. Then the diagonalising matrices are determined 
only up to an iV s -dimensional unitary redefinition of the remaining mass- 
less generations. The main result remains: if a modular symmetry mixed 
massless and heavy fields, then there would necessarily be T-dependent 
off-diagonal mass terms, which could not be set to zero except at isolated 
values of T, hence we could not actually be in the basis of light and heavy 
fields. 



c The invariant functions rrii may be complex, depending on the structure of M . 
d As discussed later, off-diagonal terms cannot in general be removed by redefinition. 
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Discussion 

Our result, which generalises that of 6 , may appear unexpected compared to 
the analogous situation for continuous gauge symmetry, where for example 
in the case of SU(2)w/ the mass eigenstates t and b (neglecting the CKM 
mixing) are mixed by some group elements and SU(2) appears to be ex- 
plicitly broken below the top mass. In that case, starting from an arbitrary 
Higgs v.e.v. of magnitude v, we are free to change basis by a global SU(2) 
redefinition to obtain a diagonal mass matrix (i.e. unitary gauge), and the 
group acts in an identical way in the new basis. But for SL(2, Z)t, the fixed 
point basis is special, and the group acts differently if one changes basis. 
One cannot in general redefine either T or the d by a group element to 
obtain a (block-)diagonal mass matrix: the diagonalisation requires V to be 
explicitly modulus-dependent, which affects the transformation properties 
of the mass eigenstates. 

The main lesson is that the freedom to assign MSSM fields to particular 
fixed points, and discard other states, is an illusion. For self-consistency, 
one should take the light fields to be invariant T-dependent combinations 
of twist states, as happens automatically in most cases. Modular invariance 
is thus a useful tool for checking the consistency of toy model results. More 
realistic models with three light generations, in which there is need to de- 
couple extra states, most likely would require more complicated string con- 
structions which would break part or all of the modular symmetry, uniquely 
determining the states that remain. One might argue that choosing a non- 
modular invariant set of states from a larger spectrum could mimic this 
situation, but the Yukawa couplings are unlikely to take the same form in 
the more realistic case. 
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